Sensory stimulation system for an autonomous vehicle

ABSTRACT

A sensory stimulation system for autonomous vehicle (AV) can determine a set of maneuvers of the AV. Based on each respective maneuver, the sensory stimulation system can determine a set of visual stimulation outputs to provide a passenger of the AV with visual indications of the respective maneuver. The sensory stimulation system can then display the set of visual stimulation outputs on a display unit within the interior of the AV prior to executing each respective maneuver.

CROSS RFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. patent application Ser. No.15/059,493, entitled “Sensory Stimulation System for an AutonomousVehicle,” and filed on Mar. 3, 2016; hereby incorporated by reference inits entirety.

BACKGROUND

With the advent of autonomous vehicle (AV) technology, rider attentionmay be focused on alternative activities, such as work, socializing,reading, writing, task-based activities (e.g., organization, billpayments, online shopping, gameplay), and the like. As the AV travelsalong an inputted route, kinetosis (i.e., motion sickness) can resultfrom the perception of motion by a rider not corresponding to therider's vestibular senses.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure herein is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements, and in which:

FIG. 1 is a block diagram illustrating an example control system foroperating an autonomous vehicle including a sensory stimulation system,as described herein;

FIG. 2 is a block diagram illustrating an example autonomous vehicleincluding a sensory stimulation system, as described herein;

FIG. 3 is a block diagram illustrating an example sensory stimulationsystem, as shown and described herein;

FIG. 4 shows an example of an autonomous vehicle utilizing sensor datato navigate an environment in accordance with example implementations;

FIGS. 5A and 5B are flow charts describing example methods of operatinga sensory stimulation system in accordance with example implementations;and

FIG. 6 is a block diagram illustrating a computer system upon whichexamples described herein may be implemented.

DETAILED DESCRIPTION

A sensory stimulation system is disclosed that provides sensorystimulation outputs responsive to or preemptive of maneuvers by theautonomous vehicle (AV). The sensory stimulation system can include anumber of output devices to provide visual, audio, tactile, and/or anycombination of vestibular stimulation for AV riders to counter,reactively and/or proactively, the sensory effects cause by AV motionthat can potentially lead to kinetosis. In some examples, the sensorystimulation system can provide a sub-conscious or unconscious learningatmosphere within the passenger interior of the AV that can preventmotion sickness and/or provide preemptive feedback to riders such thathabitual vestibular responses by the riders to outputted stimulationscan be developed. Accordingly, such habitual vestibular responses cancreate a learned correlation between the rider's vestibular system(e.g., (i) the rider's semi-circular canal system which sensesrotational movement and (ii) the rider's inner ear otoliths whichindicate linear accelerations) and the rider's visual perception.

According to examples described herein, the sensory stimulation systemcan dynamically determine maneuvers to be performed by the AV. Suchmaneuvers can include acceleration, braking, or directional changemaneuvers. In many aspects, the sensory stimulation system can receiveinputs indicating the AV's current speed, a current route traveled bythe AV, and an immediate action plan (e.g., indicating immediate actionsto be performed by the AV, such as changing lanes or braking).Additionally or alternatively, the sensory stimulation system caninclude a number of sensors, such as an accelerometer and/or gyroscopicsensor, to reactively determine the maneuvers of the AV. For eachmaneuver, the sensory stimulation system can generate a set of sensorystimulation outputs to provide a rider of the AV with sensoryindications of the maneuver, and output the set of sensory stimulationoutputs via the output devices within the interior of the AV. In variousimplementations, the sensory stimulation outputs can be generated andoutputted by the sensory stimulation system dynamically as the AVmaneuvers along a current route to a destination.

In many examples, the output devices can include visually perceptivedevices, such as a light bar visible within an interior of the AV. Thelight bar can be included to circumscribe at least a portion of theinterior passenger compartment (e.g., around the ceiling of theinterior, and/or around a mid-plane just below the windows) and canprovide visual stimulation based on each acceleration, braking, orchange of direction action performed by the AV, or combination thereof.The light bar can include multi-colored light elements which can becontrolled by the sensory stimulation system to dynamically generatecolors and brightness for respective portions of the light bar toindicate each of the maneuvers.

Additionally or alternatively, the output devices can include a numberof display units visible within the interior of the AV. For example, oneor more display units can be provided on the dashboard of the AV andbehind the front seats to provide each passenger with a view of aparticular display. In certain implementations, the sensory stimulationsystem can dynamically generate a displayed presentation that indicatesthe AV's planned actions. In some examples, the sensory stimulationsystem can generate a third-person perspective, visual representation ofthe AV traveling along a current route for display, and can furthergenerate preemptive and/or dynamic visual indications of each maneuverto be performed by the AV. In some examples, the preemptive visualindications can have a granularity that aligns with decision-makingperformed by a control system of the AV, as described herein.

In many examples, the output devices can include controllable seats. Thecontrollable seats can be operable by the sensory stimulation system andcan include haptic functionality. Additionally or alternatively, each ofthe controllable seats can include a number of motors that can controlpitch, roll, and/or yaw of the seat. As the AV travels along the currentroute, the sensory stimulation system can operate the controllable seatsto provide haptic stimulation and/or control the principal axes (i.e.,pitch, roll, yaw) of the seats based on the maneuvers of the AV.

Additionally or alternatively, the output devices of the sensorystimulation system can include an airflow system capable of providingair pressure outputs as sensory stimulations based on the maneuvers ofthe AV. The airflow system can include the AV's manufacturer installedair conditioning system and/or a customized air flow control system thatcan provide air flow stimulation to the riders from multiple directionsand at differing intensities. For example, when the AV is about tobrake, the sensory stimulation system can utilize the airflow system tomodify airflow within the cabin (e.g., change from rearward airflow toforward airflow). Airflow adjustment parameters for the airflow systemcan include airflow speed/intensity, direction (e.g., 360 degrees aroundthe riders both radially and azimuthally), temperature, timing, pulserate, and height (e.g., aiming at the rider's head, shoulders, torso,arms, legs, feet, etc.). In some implementations, airflow outlets can beprovided through the dashboard, the dashboard underside, the floor, theseats (e.g., through outlets on the headrest), or via ducting throughthe AV's chassis with outlets on the side-posts and/or doors.

Among other benefits, the examples described herein achieve a technicaleffect of providing sensory stimulation for AV riders based on themaneuvers of the AV. Such stimulation can train the sensory responses ofriders to prevent kinetosis due to uncorrelated vestibular versus visualperception.

As used herein, a computing device refers to devices corresponding todesktop computers, cellular devices or smartphones, personal digitalassistants (PDAs), laptop computers, tablet devices, television (IPTelevision), etc., that can provide network connectivity and processingresources for communicating with the system over a network. A computingdevice can also correspond to custom hardware, in-vehicle devices, oron-board computers, etc. The computing device can also operate adesignated application configured to communicate with the networkservice.

One or more examples described herein provide that methods, techniques,and actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmatically,as used herein, means through the use of code or computer-executableinstructions. These instructions can be stored in one or more memoryresources of the computing device. A programmatically performed step mayor may not be automatic.

One or more examples described herein can be implemented usingprogrammatic modules, engines, or components. A programmatic module,engine, or component can include a program, a sub-routine, a portion ofa program, or a software component or a hardware component capable ofperforming one or more stated tasks or functions. As used herein, amodule or component can exist on a hardware component independently ofother modules or components. Alternatively, a module or component can bea shared element or process of other modules, programs or machines.

Some examples described herein can generally require the use ofcomputing devices, including processing and memory resources. Forexample, one or more examples described herein may be implemented, inwhole or in part, on computing devices such as servers, desktopcomputers, cellular or smartphones, personal digital assistants (e.g.,PDAs), laptop computers, printers, digital picture frames, networkequipment (e.g., routers) and tablet devices. Memory, processing, andnetwork resources may all be used in connection with the establishment,use, or performance of any example described herein (including with theperformance of any method or with the implementation of any system).

Furthermore, one or more examples described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown or described with figures below provide examplesof processing resources and computer-readable mediums on whichinstructions for implementing examples disclosed herein can be carriedand/or executed. In particular, the numerous machines shown withexamples of the invention include processors and various forms of memoryfor holding data and instructions. Examples of computer-readable mediumsinclude permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units, such as CD or DVD units, flashmemory (such as carried on smartphones, multifunctional devices ortablets), and magnetic memory. Computers, terminals, network enableddevices (e.g., mobile devices, such as cell phones) are all examples ofmachines and devices that utilize processors, memory, and instructionsstored on computer-readable mediums. Additionally, examples may beimplemented in the form of computer-programs, or a computer usablecarrier medium capable of carrying such a program.

Numerous examples are referenced herein in context of an autonomousvehicle (AV). An AV refers to any vehicle which is operated in a stateof automation with respect to steering and propulsion. Different levelsof autonomy may exist with respect to AVs. For example, some vehiclesmay enable automation in limited scenarios, such as on highways,provided that drivers are present in the vehicle. More advanced AVsdrive without any human assistance from within or external to thevehicle. Such vehicles often are required to make advance determinationsregarding how the vehicle is behave given challenging surroundings ofthe vehicle environment.

System Description

FIG. 1 is a block diagram illustrating an example control system 100 foroperating an autonomous vehicle (AV) 10 including a sensory stimulationsystem, as described herein. In an example of FIG. 1, a control system100 can be used to autonomously operate the AV 10 in a given geographicregion for a variety of purposes, including transport services (e.g.,transport of humans, delivery services, etc.). In examples described, anautonomously driven vehicle can operate without human control. Forexample, in the context of automobiles, an autonomously driven vehiclecan steer, accelerate, shift, brake and operate lighting components.Some variations also recognize that an autonomous-capable vehicle can beoperated either autonomously or manually.

In one implementation, the control system 100 can utilize specificsensor resources in order to intelligently operate the vehicle 10 inmost common driving situations. For example, the control system 100 canoperate the vehicle 10 by autonomously steering, accelerating, andbraking the vehicle 10 as the vehicle progresses to a destination. Thecontrol system 100 can perform vehicle control actions (e.g., braking,steering, accelerating) and route planning using sensor information, aswell as other inputs (e.g., transmissions from remote or local humanoperators, network communication from other vehicles, etc.).

In an example of FIG. 1, the control system 100 includes a computer orprocessing system which operates to process sensor data that is obtainedon the vehicle with respect to a road segment upon which the vehicle 10operates. The sensor data can be used to determine actions which are tobe performed by the vehicle 10 in order for the vehicle 10 to continueon a route to a destination. In some variations, the control system 100can include other functionality, such as wireless communicationcapabilities, to send and/or receive wireless communications with one ormore remote sources. In controlling the vehicle 10, the control system100 can issue instructions and data, shown as commands 85, whichprogrammatically controls various electromechanical interfaces of thevehicle 10. The commands 85 can serve to control operational aspects ofthe vehicle 10, including propulsion, braking, steering, and auxiliarybehavior (e.g., turning lights on). In examples described herein, thecommands 85 can further serve to control output devices of a sensorystimulation system, such as visual, audio, haptic/tactile, and/orairflow output devices to provide sensory stimulation to passengers ofthe AV 10.

The AV 10 can be equipped with multiple types of sensors 101, 103, 105,which combine to provide a computerized perception of the space andenvironment surrounding the vehicle 10. Likewise, the control system 100can operate within the AV 10 to receive sensor data from the collectionof sensors 101, 103, 105, and to control various electromechanicalinterfaces for operating the vehicle on roadways.

In more detail, the sensors 101, 103, 105 operate to collectively obtaina complete sensor view of the vehicle 10, and further to obtainsituational information proximate to the vehicle 10, including anypotential hazards proximate to the vehicle 10. By way of example, thesensors 101, 103, 105 can include multiple sets of cameras sensors 101(video camera, stereoscopic pairs of cameras or depth perceptioncameras, long range cameras), remote detection sensors 103 such asprovided by radar or LIDAR, proximity or touch sensors 105, and/or sonarsensors (not shown).

Each of the sensors 101, 103, 105 can communicate with the controlsystem 100 utilizing a corresponding sensor interface 110, 112, 114.Each of the sensor interfaces 110, 112, 114 can include, for example,hardware and/or other logical component which is coupled or otherwiseprovided with the respective sensor. For example, the sensors 101, 103,105 can include a video camera and/or stereoscopic camera set whichcontinually generates image data of an environment of the vehicle 10. Asan addition or alternative, the sensor interfaces 110, 112, 114 caninclude a dedicated processing resource, such as provided with a fieldprogrammable gate array (“FPGA”) which can, for example, receive and/orprocess raw image data from the camera sensor.

In some examples, the sensor interfaces 110, 112, 114 can include logic,such as provided with hardware and/or programming, to process sensordata 99 from a respective sensor 101, 103, 105. The processed sensordata 99 can be outputted as sensor data 111. As an addition orvariation, the control system 100 can also include logic for processingraw or pre-processed sensor data 99.

According to one implementation, the vehicle interface subsystem 90 caninclude or control multiple interfaces to control mechanisms of thevehicle 10. The vehicle interface subsystem 90 can include a propulsioninterface 92 to electrically (or through programming) control apropulsion component (e.g., an accelerator pedal), a steering interface94 for a steering mechanism, a braking interface 96 for a brakingcomponent, and a lighting/auxiliary interface 98 for exterior lights ofthe vehicle. According to implementations described herein, controlsignals 119 can be transmitted to a stimulation interface 95 of thevehicle interface subsystem 90 to control sensory stimulation outputsthrough various output devices of a sensory stimulation system of the AV10. The vehicle interface subsystem 90 and/or the control system 100 canfurther include one or more controllers 84 which can receive commands 85from the control system 100. The commands 85 can include routeinformation 87 and operational parameters 89—which specify anoperational state of the vehicle 10 (e.g., desired speed and pose,acceleration, etc.)—as well as stimulation commands 85 to control theoutput devices of the sensory stimulation system.

The controller(s) 84 can generate control signals 119 in response toreceiving the commands 85 for one or more of the vehicle interfaces 92,94, 95, 96, 98. The controllers 84 can use the commands 85 as input tocontrol propulsion, steering, braking, and/or other vehicle behaviorwhile the AV 10 follows a current route. Thus, while the vehicle 10actively drives along the current route, the controller(s) 84 cancontinuously adjust and alter the movement of the vehicle 10 in responseto receiving a corresponding set of commands 85 from the control system100. Absent events or conditions which affect the confidence of thevehicle 10 in safely progressing along the route, the control system 100can generate additional commands 85 from which the controller(s) 84 cangenerate various vehicle control signals 119 for the differentinterfaces of the vehicle interface subsystem 90.

According to examples, the commands 85 can specify actions to beperformed by the vehicle 10. The actions can correlate to one ormultiple vehicle control mechanisms (e.g., steering mechanism, brakes,etc.). The commands 85 can specify the actions, along with attributessuch as magnitude, duration, directionality, or other operationalcharacteristics of the vehicle 10. By way of example, the commands 85generated from the control system 100 can specify a relative location ofa road segment which the AV 10 is to occupy while in motion (e.g.,change lanes, move into a center divider or towards shoulder, turnvehicle, etc.). As other examples, the commands 85 can specify a speed,a change in acceleration (or deceleration) from braking or accelerating,a turning action, or a state change of exterior lighting or othercomponents. The controllers 84 can translate the commands 85 intocontrol signals 119 for a corresponding interface of the vehicleinterface subsystem 90. The control signals 119 can take the form ofelectrical signals which correlate to the specified vehicle action byvirtue of electrical characteristics that have attributes for magnitude,duration, frequency or pulse, or other electrical characteristics.

In an example of FIG. 1, the control system 100 can include a routeplanner 122, stimulation logic 121, event logic 124, and a vehiclecontrol 128. The vehicle control 128 represents logic that convertsalerts of event logic 124 (“event alert 135”) and sensory outputs 133 bythe stimulation logic 121 into commands 85 that specify a set of vehicleactions and/or sensory stimulation outputs.

Additionally, the route planner 122 can select one or more routesegments that collectively form a path of travel for the AV 10 when thevehicle 10 is on a current trip (e.g., servicing a pick-up request). Inone implementation, the route planner 122 can specify route segments 131of a planned vehicle path which defines turn by turn directions for thevehicle 10 at any given time during the trip. The route planner 122 mayutilize the sensor interface 110 to receive GPS information as sensordata 111. The vehicle control 128 can process route updates from theroute planner 122 as commands 85 to progress along a path or route usingdefault driving rules and actions (e.g., moderate steering and speed).

According to examples described herein, the control system 100 canfurther execute stimulation logic 121 to provide sensory outputs 133 tothe vehicle control 128 based on maneuvers performed by the AV 10. Insome aspects, the stimulation logic 121 can utilize route information(e.g., indicating a granular path to be traveled by the AV 10),situational data (e.g., indicating the surrounding entities, road signs,traffic signals, etc. proximate to the AV 10), and/or action informationindicating the planned actions to be performed by the control system 100in maneuvering the AV 10. Based on the foregoing collective data, thestimulation logic 121 can generate sensory outputs 133 corresponding tosensory stimulations to be provided to riders of the AV 10 based onanticipated or current maneuvers of the AV 10. Such maneuvers caninclude acceleration, braking, and directional change maneuvers withrelatively fine granularity. For example, the maneuvers can includeminor swerves, light braking and acceleration, and low speed turns aswell as harder braking and acceleration and normal to aggressive turns.Additionally or alternatively, the stimulation logic 121 can utilizesensor data 111 indicating acceleration or inertial information (e.g.,specific force or angular rate) in order to generate the sensory outputs133.

The sensory outputs 133 can be processed by the vehicle control 128 togenerate stimulation commands 85, which the controller 84 can process tooperate interior cabin stimulation systems of the AV 10. For example,based on the sensory commands 85, the controller 84 can generate controlsignals 119 to engage the interior visual, audio, haptic/tactile,airflow, and seat positioning systems via the stimulation interface 95in order to provide the rider(s) with sensory stimulation tosubstantially correlate vestibular perception with visual perception.Such stimulation outputs can provide sufficient correlation or agreementbetween the vestibular sense of movement and visual perception toprevent kinetosis and its unpleasant symptoms. Detailed description ofthe sensory stimulation system is provided below with respect to FIGS. 2through 5B.

In certain implementations, the event logic 124 can trigger a responseto a detected event. A detected event can correspond to a roadwaycondition or obstacle which, when detected, poses a potential hazard orthreat of collision to the vehicle 10. By way of example, a detectedevent can include an object in the road segment, heavy traffic ahead,and/or wetness or other environmental conditions on the road segment.The event logic 124 can use sensor data 111 from cameras, LIDAR, radar,sonar, or various other image or sensor component sets in order todetect the presence of such events as described. For example, the eventlogic 124 can detect potholes, debris, objects projected to be on acollision trajectory, and the like. Thus, the event logic 124 can detectevents which enable the control system 100 to make evasive actions orplan for any potential threats.

When events are detected, the event logic 124 can signal an event alert135 that classifies the event and indicates the type of avoidance actionto be performed. Additionally, the control system 100 can determinewhether an event corresponds to a potential incident with a human drivenvehicle, a pedestrian, or other human entity external to the AV 10. Inturn, the vehicle control 128 can determine a response based on thescore or classification. Such response can correspond to an eventavoidance action 145, or an action that the vehicle 10 can perform tomaneuver the vehicle 10 based on the detected event and its score orclassification. By way of example, the vehicle response can include aslight or sharp vehicle maneuvering for avoidance using a steeringcontrol mechanism and/or braking component. The event avoidance action145 can be signaled through the commands 85 for controllers 84 of thevehicle interface subsystem 90.

When an anticipated dynamic object of a particular class does in factmove into position of likely collision or interference, some examplesprovide that event logic 124 can signal the event alert 135 to cause thevehicle control 128 to generate commands 85 that correspond to an eventavoidance action 145. For example, in the event of a bicycle crash inwhich the bicycle (or bicyclist) falls into the path of the vehicle 10,event logic 124 can signal the event alert 135 to avoid the collision.The event alert 135 can indicate (i) a classification of the event(e.g., “serious” and/or “immediate”), (ii) information about the event,such as the type of object that generated the event alert 135, and/orinformation indicating a type of action the vehicle 10 should take(e.g., location of object relative to path of vehicle, size or type ofobject, etc.). In addition, the stimulation logic 121 can utilize theevent alert 135 to cause the controller 84 to generate a correspondingoutput via the stimulation interface 95 to provide the rider withsensory stimulation to anticipate or react to the event.

FIG. 2 is a block diagram illustrating an example AV 200 including asensory stimulation system 235, as described herein. The AV 200 shown inFIG. 2 can include some or all aspects and functionality of the AV 10described with respect to FIG. 1. Referring to FIG. 2, the AV 200 caninclude a sensor array 205 that can provide sensor data 207 to anon-board data processing system 210. As described herein, the sensorarray 205 can include any number of active or passive sensors thatcontinuously detect a situational environment of the AV 200. Forexample, the sensor array 205 can include a number of camera sensors(e.g., stereoscopic cameras), LIDAR sensor(s), proximity sensors, radar,and the like. The data processing system 210 can utilize the sensor data207 to detect the situational conditions of the AV 200 as the AV 200travels along a current route. For example, the data processing system210 can identify potential obstacles or road hazards—such aspedestrians, bicyclists, objects on the road, road cones, road signs,animals, etc.—in order to enable an AV control system 220 to reactaccordingly.

In certain implementations, the data processing system 210 can utilizesub-maps 231 stored in a database 230 of the AV 200 (or accessedremotely from the backend system 290 via the network 280) in order toperform localization and pose operations to determine a current locationand orientation of the AV 200 in relation to a given region (e.g., acity).

The data sub-maps 231 in the database 230 can comprise previouslyrecorded sensor data, such as stereo camera data, radar maps, and/orpoint cloud LIDAR maps. The sub-maps 231 can enable the data processingsystem 210 to compare the sensor data 207 from the sensor array 205 witha current sub-map 238 to identify obstacles and potential road hazardsin real time. The data processing system 210 can provide the processedsensor data 213—identifying such obstacles and road hazards—to the AVcontrol system 220, which can react accordingly by operating thesteering, braking, and acceleration systems 225 of the AV 200 to performlow level maneuvering.

In many implementations, the AV control system 220 can receive adestination 219 from, for example, an interface system 215 of the AV200. The interface system 215 can include any number of touch-screens,voice sensors, mapping resources, etc., that enable a passenger 239 toprovide a passenger input 241 indicating the destination 219. Forexample, the passenger 239 can type the destination 219 into a mappingengine 275 of the AV 200, or can speak the destination 219 into theinterface system 215. Additionally or alternatively, the interfacesystem 215 can include a wireless communication module that can connectthe AV 200 to a network 280 to communicate with a backend transportarrangement system 290 to receive invitations 282 to service a pick-upor drop-off request. Such invitations 282 can include the destination219 (e.g., a pick-up location), and can be received by the AV 200 as acommunication over the network 280 from the backend transportarrangement system 290. In many aspects, the backend transportarrangement system 290 can manage routes and/or facilitatetransportation for users using a fleet of autonomous vehicles throughouta given region. The backend transport arrangement system 290 can beoperative to facilitate passenger pick-ups and drop-offs to generallyservice pick-up requests, facilitate delivery such as packages or food,and the like.

Based on the destination 219 (e.g., a pick-up location), the AV controlsystem 220 can utilize the mapping engine 275 to receive route data 232indicating a route to the destination 219. In variations, the mappingengine 275 can also generate map content 226 dynamically indicating theroute traveled to the destination 219. The route data 232 and/or mapcontent 226 can be utilized by the AV control system 220 to maneuver theAV 200 to the destination 219 along the selected route. For example, theAV control system 220 can dynamically generate control commands 221 forthe autonomous vehicle's steering, braking, and acceleration systems 225to actively drive the AV 200 to the destination 219 along the selectedroute.

In many examples, while the AV control system 220 operates the steering,braking, and acceleration systems 225 along the current route on a highlevel, the processed data 213 provided to the AV control system 220 canindicate low level occurrences, such as obstacles and potential hazards,to which the AV control system 220 can make decisions and react. Forexample, the processed data 213 can indicate a pedestrian crossing theroad, traffic signals, stop signs, other vehicles, road conditions,traffic conditions, bicycle lanes, crosswalks, pedestrian activity(e.g., a crowded adjacent sidewalk), and the like. The AV control system220 can respond to the processed data 213 by generating control commands221 to reactively operate the steering, braking, and accelerationsystems 225 accordingly.

According to examples described herein, the AV 200 can include a sensorystimulation system 235 that can operate a number of interior outputsystems 240 based on motion actions or maneuvers of the AV 200. Thesensory stimulation system 235 can provide such stimulation outputsreactively in response to AV maneuvers, or preemptively in anticipationof such maneuvers. In many examples, the sensory stimulation system 235in combination with the interior output systems 240 can provide asubconscious or unconscious learning environment for correlatingvestibular sensing with visual perception in order to prevent motionsickness or unexpected surprises when the AV 200 performs ordinary oremergency maneuvers.

In many examples, the sensory stimulation system 235 can dynamicallyreceive an action plan 229 from the AV control system 220 indicating theimmediate maneuvers to be performed by the AV control system 220. Forexample, the action plan 229 can correspond to the control commands 221that the AV control system 220 generates to operate the steering,braking, and acceleration systems 225 of the AV 200. Thus, the actionplan 229 can indicate low level inputs to be provided to, for example,an accelerator, brake, or steering mechanism of the AV 200 as the AV 200is autonomously operated along a current route. The sensory stimulationsystem 235 can process the action plan 229 to generate stimulationcommands 233 for the interior output systems 240—such as visualstimulation, dynamic seat adjustments (e.g., adjusting pitch, roll,and/or yaw), haptic stimulation, and/or air pressure stimulation.

Additionally or alternatively, the sensory stimulation system 235 canreceive situational data 217 from the on-board data processing system210. The situational data 217 can include data identifying externalentities and their locations proximate to the AV 200. External entitiescan include any proximate pedestrians, bicyclists, human-drivenvehicles, or any other human within proximity of the AV 200. Accordingto an example implementation, the interior output systems 240 caninclude one or more displays, such as a large dash display above acenter console area of the AV 200, or multiple dash displays prominentlyvisible by each of the passengers. The sensory stimulation system 235can dynamically generate a live map view of the immediate environment ofthe AV 200 for presentation on the displays. For example, the sensorystimulation system 235 can continuously present a three-quarterperspective or pseudo-three-dimensional perspective view from above andbehind the AV 200, and dynamically generate virtual representations ofthe AV 200 as well as each of the proximate external entities identifiedin the situational data 217. Thus, as the AV 200 travels, the displayedpresentation can be dynamically updated to show a live view of the AV200 itself traveling along a current route and representations ofexternal entities identified by the sensory stimulation system 235 inthe situational data 217.

In some aspects, the sensory stimulation system 235 can present actuallive video and/or LIDAR data on the display(s) that indicate an entiresituational environment of the AV 200 (e.g., in a forward operationalfield of view of the AV 200). In such aspects, the sensory stimulationsystem 235 can generate augmented reality content to superimpose certainfeatures in the live presentation, and/or present augmented realityindicators that provide information regarding the route to be traveledby the AV 200 and low level maneuvers to be performed by the AV 200, asdescribed herein.

In variations, the sensory stimulation system 235 can further receiveroute data 232 indicating the current route traveled by the AV 200 tothe destination 219. On a coarse granular level, the sensory stimulationsystem 235 can generate route indicators on the presented live map basedon the route data 232, where the route indicators can highlight thecurrent route traveled by the AV 200. On a finer granular level, thesensory stimulation system 235 can utilize the action plan 229 from theAV control system 220—as well as the route data 232 and situational data217—to generate low level indications of the AV 200 immediate actionsfor the presented display. For example, the action plan 229 can identifysubtle actions to be performed by the AV control system 220, such asindividual lane changes, yielding actions, planned braking andacceleration actions, avoidance maneuvers, and even emergency maneuvers.The sensory stimulation system 235 can utilize this low level actionplan 229 data to generate preemptive indicators on the dynamicallydisplayed presentation of each of the AV's 200 maneuvers. Such low levelindicators can include generated symbols and/or arrows on the live mapview (e.g., color coded arrows to indicate acceleration, braking, andturning actions), audio speech preemptively describing the actions ormaneuvers, audio sounds indicating each maneuver (e.g., jingles or tonesfor each respective action), or displayed words describing the actions.

Utilizing each of the situational data 217, the route data 232, and theaction plan 229 from the AV control system 235, the sensory stimulationsystem 235 can dynamically generate stimulation commands 233 tocontinuously stream the live presentation via the display(s) of theinterior output systems 240. The live presentation can preemptivelyprovide the passenger(s) with each low level maneuver to be performed bythe AV control system 220, as well as a route plan from the route data232 and representations of proximate external entities from thesituational data 217. Utilizing the actual control data of the AVcontrol system 220, the live presentation can provide far more granulardetail of the AV's 200 route and actions than currently available liveviews, which typically provide only a cursory route plan. Furthermore,the sensory stimulation system 235 can provide the live presentation asa standalone visual stimulation for the passenger(s), or in conjunctionwith other sensory stimulations utilizing various output devices of theinterior output systems 240.

In addition or as an alternative to the above live presentation, thesensory stimulation system 235 can further utilize the action plan data229 to generate stimulation commands 233 for a fixed lighting system ofthe interior output systems 240. In some examples, the fixed lightingsystem can comprise a light bar that includes a string of lightingelements (e.g., RGB LEDS) circumscribing the interior of the AV 200. Forexample, the light bar can circumscribe an edge of the ceiling of the AV200. Additionally or alternatively, the light bar can circumscribe amid-plane of the AV interior (e.g., just below the windows). Accordingto one example, the interior output systems 240 can include multiplelight bars or lighting elements fixed within the interior of the AV 200.The sensory stimulation system 235 can utilize the action plan 229 togenerate stimulation commands 233 that cause the lighting system tooutput visual sensory indications of the AV's 200 upcoming and/orcurrent maneuvers.

For example, the sensory stimulation system 235 can generate stimulationcommands 233 controlling lighting outputs that indicate eachacceleration, braking, and directional change maneuver to be performedby the AV control system 220. The stimulation commands 233 can triggercertain portions of the lighting system (e.g., illuminating the leftside of a light bar to indicate an upcoming or current left turningmaneuver), control color and brightness or a blink rate (e.g., toindicate acceleration versus braking, and the intensities of suchmaneuvers and actions). In certain aspects, the sensory stimulationsystem 235 can provide lighting outputs to coincide with theacceleration, braking, and steering systems 225, such that thevestibular and visual perception of passengers may be readily trained toanticipate or otherwise react to each maneuver, however minor.

As an example, acceleration may be correlated with green, braking withred, and turning with yellow. As the AV 200 travels, the sensorystimulation system 235 can generate stimulation commands 233 that causethe lighting system of the interior output systems 240 to preemptivelyand/or conjunctively illuminate the lighting system to indicateacceleration, braking, and turning. Furthermore, the sensory stimulationsystem 235 can control brightness to indicate an intensity or strengthof each acceleration, braking, and turning maneuver, as well asilluminating select portions of the lighting system to indicatedirectional aspects of the maneuvering. The sensory stimulation system235 can generate such stimulation commands 233 dynamically to providepassengers with visual indications or feedback of the AV's 200movements.

Additionally or alternatively, the sensory stimulation system 235 canutilize the action plan data 229, as well as the situational data 217and route data 232 to provide further stimulation to riders, such ashaptic or tactile stimulation, air pressure stimulation, and/or rotatingthe seats on principal axes to provide perceived force to thepassenger(s). Examples and detailed explanation of such stimulations bythe sensory stimulation system 235 is provided in the below descriptionof FIG. 3.

FIG. 3 is a block diagram illustrating an example sensory stimulationsystem 300 as shown and described herein. The sensory stimulation system300 shown and described with respect to FIG. 3 can include some or allof the functionality of the sensory stimulation system 235 and theinterior output systems 240 discussed above with respect to FIG. 2.Referring to FIG. 3, the sensory stimulation system 300 can include amemory 350 that stores correlation logs 352 providing correlationsbetween motion actions and maneuvers of the AV 200 with stimulation sets354 that can be executed to provide riders with sensory stimulation viathe output systems 390. Furthermore, the sensory stimulation system 300can include a stimulation engine 320 that can process situational data314, route data 311, and/or action data 313 provided from the AV controlsystem 305.

The sensory stimulation system 300 can include an AV interface 310 toreceive control system data 309 from the AV control system 305. In someaspects, the control system data 309 can include the action data 313indicating the immediate actions or maneuvers to be executed by the AVcontrol system 305. Additionally, the control system data 309 caninclude situational data 314 and/or route data 311, as described above.The control system data 309 can be processed by a stimulation engine 320of the sensory stimulation system 300. Based on the control system data309, the stimulation engine 320 can determine a number of sensorystimulation outputs to generate via the output systems 390.

As described herein, the output systems 390 of the sensory stimulationsystem 300 can include a visual system 393 that can comprise thedisplay(s) and/or lighting system (e.g., light bar(s) circumscribing theinterior of the AV). Utilizing the control system data 309 (e.g., thesituational data 314, route data 311, and/or action data 313), thestimulation engine 320 can generate stimulation commands 322 to providea live presentation of the AV, described above, and/or lighting outputsthat indicate each of the maneuvers of the AV, also described above,using the visual system 393.

In addition or as an alternative to the foregoing aspects, the sensorystimulation system 300 can utilize the control system data 309 togenerate stimulation commands 322 for other output systems 390, such asthe audio system 395, a seat response system 397, and/or an airflowsystem 399. The audio system 395 can be implemented with the AV'smanufacturer installed speaker sets or as an independent speaker set. Insome aspects, the audio system 395 can be utilized in conjunction withother output devices of the sensory stimulation system 300 to indicateinstant maneuvers to be performed by the AV (e.g., via output jingles,directional tones, or speech).

The seat response system 397 can include haptic seat technology that canprovide vibrational or other tactile feedback to the passenger via theseats. For example, haptic seats can provide vibrational pulses to thewhole seat, or select portions of the seat based on the maneuver (e.g.,left side or right side to indicate turns, upper portion to indicateacceleration, and forward seat cushion to indicate braking).Furthermore, the sensory stimulation system 300 can vary an intensity orstrength of the haptic feedback and/or vibrational pulses based on anintensity or strength of the maneuver.

In certain aspects, the seat response system 397 can include a number ofmotors and a pivoting mechanism for each seat to enable the stimulationengine 320 to adjust a pitch, roll, and/or yaw of each seat in responseto the control system data 309. For example, the action data 313 fromthe AV control system 305 can indicate an upcoming sharp left turn(e.g., to be performed within one or two seconds). In response, thestimulation engine 320 can generate a stimulation command 322 to rollthe seats to the left to direct the force felt by the rider downward(into the seat cushion) as opposed to laterally based on the centripetalforce of the AV as it makes the turn. For acceleration and braking, thestimulate engine 320 can generate stimulation commands 322 causing theseats to pitch forward or backward. In some aspects, the stimulationengine 320 can further control the yaw of the seats, for example, whenthe AV performs turning maneuvers or combinations of turning andacceleration or braking. Additionally or alternatively, based on theaction data 313, the stimulation engine 320 can generate combinations ofpitch, roll, and yaw stimulation commands 322 for the seat responsesystem 397 dynamically as the AV travels along a current route.

The airflow system 399 can output directional airstreams to provideriders with stimulation indicating movement of the AV. As providedherein, the airflow system 399 can utilize the air conditioning systemof the AV, or can be provided as a customized system capable ofproviding directional airstreams to the rider(s). The stimulation engine320 can operate compressed air valves and/or fans that can generate airpulses or wind that can indicate a direction of travel, speed,acceleration, braking, and changes in direction. The airflow system 399can include various outlets to provide the airstreams from multipledirections. For example, the airflow system 399 can include air ductingand outlets through the dash (or utilize existing ducts of the AV),through the doors, the ceiling, and/or the seats of the AV. As anexample, the airflow system 399 can include outlets on the headrests ofthe passenger seats to provide airstream outputs or an air pulse whenthe AV performs a specified action (e.g., performs an avoidancemaneuver). As another example, the stimulation engine 320 can cause theairflow system 399 to provide simulated wind to provide the riders withairflow stimulation indicating that the AV is in motion.

The stimulation engine 320 can control various parameters of the airflowsystem 399 based on the action data 313, such as airflowspeed/intensity, direction (e.g., 360° around the riders both radiallyand azimuthally), temperature, timing, pulse rate, and height (e.g.,aiming at the rider's head, shoulders, torso, arms, legs, feet, etc.).In many aspects, the stimulation engine 320 can generate stimulationcommands 322 to utilize the airflow system 399 individually, or inconjunction with the other output devices 390 of the sensory stimulationsystem 300.

Additionally, each of the output systems 390 may be activated ordeactivated by the rider. Accordingly, if the rider wishes todeactivate, for example, the seat response system 397, the rider canprovide input to do so. For example, the display(s) of the visual system393 can include a user interface to enable the rider to access a menuindicating the various stimulation outputs of the sensory stimulationsystem 300. The rider may activate or deactivate any one or all of theoutput systems 390 as desired. In certain examples, the rider cancontrol a sensitivity of the output systems 390 to increase or decreasestimulation responses. For example, a rider may wish to disable theaudio system 395 of the sensory stimulation system 300 in order tolisten to music or the radio. As another example, the rider may wish todisable the live presentation to view a news or entertainment program.Accordingly, the rider may adjust or disable any one or more of theoutput systems 390 to his or her liking.

According to examples described herein, the AV interface 310 can receivecontrol system data 309 from the AV control system 305, and parse thedata into various aspects, such as situational data 314 indicating thesituational environment and external entities, route data 311 indicatinga current route traveled by the AV to a destination, and action data 313corresponding to immediate actions to be executed by the AV controlsystem 305 in autonomously operating the AV's acceleration, braking, andsteering systems. In certain examples, the sensory stimulation system300 can further include sensors 330, such as one or more accelerometersand/or gyroscopic sensors to provide force data 331 to the stimulationengine 320 to reactively generate stimulation commands 322.

In one aspect, the stimulation engine 320 can utilize the situationaldata 314 and the route data 311 to provide a live, visual presentationon displays of the visual system 393. The stimulation engine 320 canfurther utilize the action data 313 from the control system 305 todynamically generate action indicators—comprising symbols, arrows,colors, words, etc.—that indicate the actions and maneuvers beingperformed or to be performed by the AV. Thus, the displayed livepresentation can include low level granular information indicating eachparticular action the AV control system 305 performs when autonomouslyoperating the AV's acceleration, braking, and steering systems.

Additionally or alternatively, based on the action data 313, thestimulation engine 320 can perform lookups 329 in the correlation logs352 of the memory 350 to identify a selected stimulation set 356 thatmatches the maneuver(s) or action(s) to be performed. For example, theaction data 313 can indicate that the AV control system 305 is operatingthe AV at freeway speeds and will exit the freeway with relatively hardbraking to make a tight right turn. A stimulation set 354 for such acombination can be selected and utilized by the stimulation engine 320to generate a set of stimulation commands 322 for each of the visual393, audio 395, seat response 397, and airflow systems 399 to providestimulation to the rider(s) to anticipate the maneuver and compensatefor the experienced forces caused by the maneuver. For example, thestimulation commands 322 can cause the lighting elements (e.g., a lightbar) of the visual system 393 to flash red to indicate the hard braking,and then provide bright yellow outputs on a right-hand portion of thelight bar to indicate the hard right turn.

Additionally or as an alternative, the stimulation commands 322 cangenerate an audio output via the audio device 395 to provide a speechoutput or tonal audio stimulations to indicate each step of themaneuver. Additionally or alternatively still, the stimulation commands322 can cause the seat response system 397 to pitch the seat rearwardsto compensate for the forward braking forward, and then roll the seatsrightward to compensate for the turning force as the AV makes the sharpright-hand turn. As a further addition or alternative, as the AV travelsat freeway speeds, the stimulation commands 322 can cause the airflowsystem 399 to provide a gentle and continuous rearward breeze, andprovide a directional pulse of air as the AV goes under braking (e.g.,from the rear of the rider), and another directional pulse as the AVturns right (e.g., from the front right side of the rider's head).

Such sensory stimulations using the output systems 390 can be generatedprimarily based on the action data 313 received from the AV controlsystem 305, which can include preemptive maneuvering information as wellas currently executing maneuver information. Thus, certain sensorystimulations can be provided prior to the maneuvers being executed(e.g., the airflow stimulations, visual stimulations using the lightbar, and haptic seat stimulations), and other sensory stimulations canbe provided as the maneuver is being performed (e.g., the seat roll andpitch response stimulations).

In variations, the force data 331 provided by the sensors 330 of thesensory stimulation system 300 can be utilized by the stimulation engine320 to reactively operate one or more of the output systems 390. Theforce data 331 can indicate directional acceleration experienced by theAV due to the AV accelerating, braking, turning, experiencing bumps orother forces on a bumpy road, and combinations thereof (e.g., swervingor sliding). As an example, the seat roll, pitch, and yaw responses ofthe seat response system 397 can be based on the force data 331 generateby the sensors 330 of the sensory stimulation system 300. In otherexamples, the stimulation engine can generate stimulation commands 322for the airflow system 399 and/or the visual system 393 based on theforce data 331 from the sensors 330.

Thus, utilizing control system data 309 from the AV control system 305and/or force data 331 from the sensors 330, the stimulation engine 320can generate stimulation commands 322 that provide visual, audible,tactile/haptic, force (e.g., via pitching and rolling the seats), and/orairflow outputs to provide vestibular stimulation to the rider(s) toprevent or mitigate the effects of kinetosis. Each of the output systems390 can be variably controlled by the stimulation engine 320 based on anintensity of a particular maneuver or motion action. Furthermore, userinput may be provided by a rider to activate, deactivate, or adjust thecontrol parameters of any or all of the output systems 390.

Autonomous Vehicle in Operation

FIG. 4 shows an example of an autonomous vehicle 410 utilizing sensordata to navigate an environment 400 in accordance with exampleimplementations. In an example of FIG. 4, the autonomous vehicle 410 mayinclude various sensors, such as a roof-top camera array (RTC) 422,front-facing cameras 424 and laser rangefinders 430. A data processingsystem 425, comprising a combination of one or more processors andmemory units, can be positioned in a trunk of the vehicle 410.

According to an example, the vehicle 410 uses one or more sensor views403 (e.g., a stereoscopic view or 3D LIDAR imaging of the environment400) to scan a road segment on which the vehicle 410 traverses. Thevehicle 410 can process image data, corresponding to the sensor views403 from one or more sensors in order to detect objects that are, or maypotentially be, in the path of the vehicle 410. In an example shown, thedetected objects include a bicyclist 402, a pedestrian 404, and anothervehicle 427—each of which may potentially cross into a road segment 415along which the vehicle 410 traverses. The vehicle 410 can useinformation about the road segment and/or image data from the sensorviews 403 to determine that the road segment includes a divider 417 andan opposite lane, a traffic signal 440, and a sidewalk (SW) 421 andsidewalk structures such as parking meters (PM) 437.

The vehicle 410 may determine the location, size, and/or distance ofobjects in the environment 400 based on the sensor view 403. Forexample, the sensor views 403 may be 3D sensor images that combinesensor data from the roof-top camera array 422, front-facing cameras424, and/or laser rangefinders 430. Accordingly, the vehicle 410 mayaccurately detect the presence of objects in the environment 400,allowing the vehicle to safely navigate the route while avoidingcollisions with other objects.

As described herein, a sensory stimulation system of the vehicle 410 candisplay a live presentation of external entities such as the bicyclist402, the pedestrian 404, and the human-driven vehicle 427 within theenvironment 400 through which the vehicle 410 travels. Furthermore,assuming that the vehicle 410 is operating at a steady speed, thevehicle 410 can utilize the sensor view 403 to identify that the trafficsignal 440 has changed from a green state to a yellow state (as shown),which can cause the control system of the vehicle 410 to generate anaction plan to apply the brakes of the vehicle 410 to decelerate thevehicle 410 at a certain rate (e.g., six meters per second per second)to come to a complete stop at the traffic signal 440. The sensorystimulation system of the vehicle 410 can utilize this action data togenerate stimulation commands in order to provide stimulation outputsfor the riders of the vehicle 410 and compensate or otherwise preparethe riders for the braking, as described herein.

According to examples, the vehicle 410 may further determine aprobability that one or more objects in the environment 400 willinterfere or collide with the vehicle 410 along the vehicle's currentpath or route. In some aspects, the vehicle 410 may selectively performan avoidance action based on the probability of collision. The avoidanceactions may include velocity adjustments, lane aversion, roadwayaversion (e.g., change lanes or driver far from curb), light or hornactions, and other actions. In some aspects, the avoidance action mayrun counter to certain driving conventions and/or rules (e.g., allowingthe vehicle 410 to drive across center line to create space withbicyclist).

For each such avoidance action, the sensory stimulation system candetermine the planned maneuver and control the interior output systems(e.g., the visual, audio, seat response, haptic, and/or airflow systems)to provide sensory stimulation for the riders based on the maneuver. Forexample, a swerve maneuver can cause the sensory stimulation system togenerate lighting, haptic, seat adjustment, and/or airflow responses toprovide the riders with vestibular stimulation according to themaneuver, as provided herein. Such responses can provide at least somecorrelation between the riders' visual perception and the forces felt asthe maneuvering is being performed.

Methodology

FIGS. 5A and 5B are flow charts describing example methods of operatinga sensory stimulation system in accordance with example implementations.In the below descriptions of FIGS. 5A and 5B, reference may be made tolike reference characters representing various features shown anddescribed with respect to FIGS. 1 through 3. Furthermore, the methodsdescribed in connection with FIGS. 5A and 5B may be performed by examplesensory stimulation systems 235, 300 shown and described with respect toFIGS. 2 and 3. Referring to FIG. 5A, the sensory stimulation system 300can monitor AV maneuvering data from the AV control system 305 (500). Inmany examples, the AV maneuvering data can be included in action data313 or an action plan 229 indicating planned, low level maneuvering tobe performed by the AV control system 305 (502). Such low level plannedmaneuvering is distinguishable from a high level route plan indicating acurrent route in that monitoring the planned maneuvering (502) canprovide the sensory stimulation system 300 with immediate actions by theAV control system 305 to be implemented in autonomously controlling theAV 200 (e.g., acceleration, steering, and braking actions). Additionallyor alternatively, the sensory stimulation system 300 can monitor activemaneuvers as they are being performed by the AV (504). For example, thesensory stimulation system 300 can receive the control commands 221 asthey are being implemented, or monitor one or more sensors, such as anaccelerometer and/or gyroscopic sensor indicating lateral forceexperienced by the AV 200.

For each maneuver, the sensory stimulation system 300 can generatestimulation commands 322 for the interior output systems 390 of the AV200 (505). The stimulation commands 322 can be generated to compensatefor anticipated lateral forces, provide sensory stimulation to “trick”or otherwise enable riders to correlate vestibular senses with visualperception, and prevent sensory disruptions that can affect ridercomfort. The sensory stimulation system 300 can generate stimulationcommands 322 for visual systems 393, such as the display(s) or lightingelements (e.g., a light bar) (506), seat response systems 397 to providehaptic feedback and/or pitch, roll, yaw sensations (507), an audiosystem 395 to provide speech and/or tonal stimulations indicating eachmaneuver (508), and/or an airflow system 399 to provide air pressurestimulation based on the maneuvers (509).

Accordingly, the sensory stimulation system 300 can dynamically executethe stimulation commands 322 on the output systems 390 to providevestibular stimulation based on each of the maneuvers (510). Executionof the stimulation commands 322 can be performed preemptively (512) toenable riders to anticipate specific maneuvers (e.g., providing visualstimulation indicating upcoming braking), or reactively (514) ascompensatory stimulation (e.g., rolling the seats during a turn, orproviding simulated wind based on acceleration). As a dynamic process,the sensory stimulation system 300 can continuously function as the AV200 is autonomously operated, the sensory stimulation system 500 canthen continuously monitor AV maneuvering data accordingly (500).

Referring to FIG. 5B, the sensory stimulation system 300 can receivesituational data 217 from the sensor array 205 of the AV 200 (515). Thesensory stimulation system 300 can dynamically identify externalentities in the situational data 217, such as bicyclists, pedestrians,other vehicles, and the like (520). The sensory stimulation system 300can also receive route data 232 indicating a current route traveled bythe AV 200 (525). Utilizing the situational data 217 and the route data232, the sensory stimulation system 300 can generate a live presentationon a number of display devices (530). The live presentation can comprisea third-person highball view from above and behind the AV 200 andinclude a live map indicating route information (532) indicating theroute traveled (e.g., a highlighted route on the live map), a virtualrepresentation of the AV 200 as it travels along the route, andgenerated representations of the external entities superimposed on thelive presentation and indicating their dynamic positions (534).Alternatively, the sensory stimulation system 300 can present livecamera data and/or LIDAR data on the display(s), with one or moreaugmented reality elements generated thereon (e.g., highlighting thecurrent route).

In many examples, the sensory stimulation system 300 can also receiveand monitor AV maneuvering data from the AV control system 305 and othermotion actions (e.g., road bumps, vertical movement on a windy road,etc.) (535). The sensory stimulation system 300 can identify plannedmaneuvers (e.g., to be executed within 5 seconds) (537) and currentmaneuvers being performed by the AV control system 305 (539). Based onthe planned and/or current maneuvers of the AV control system 305, thesensory stimulation system 300 can dynamically generate live maneuveringindicators for the displayed live presentation (540). The sensorystimulation system 300 can input or otherwise superimpose the livemaneuvering indicators, identifying planned as well as currentmaneuvers, onto the live presentation on top of the high level routeinformation and live map. Such indicators can comprise color codedand/or flashing symbols, arrows, voice output, tonal outputs, etc.,which can indicate granular, low level acceleration, turning, braking,swerving, lane changing, and other like actions.

Additionally or alternatively, the sensory stimulation system 300 canutilize the AV maneuvering data (e.g., action data 313) to dynamicallygenerate stimulation commands 322 for each planned and/or activemaneuver (545). The stimulation commands 322 can be based on maneuveringdata received from the AV control system 305 (547), or based on forcedata 331 received from sensor(s) 330 of the sensory stimulation system300 (549). In generating the stimulation commands 322, the sensorystimulation system 300 can vary output strength parameters based onmaneuvering intensity (550). Such output strength parameters can includecolor, brightness, and/or blink rate for the live presentation andlighting elements, volume or audible urgency for the audio system 395,haptic strength, location, and pulse rate for the haptic system, anglesand speed of pitch, roll, and/or yaw for the seat response system 397,and direction, pulse rate, wind speed, and/or temperature for theairflow system 399.

When the output strength parameters are configured and the stimulationcommands 322 generated, the sensory stimulation system 300 can executethe commands on the respective output systems 390 in a timed manner(555). Specifically, the sensory stimulation system 300 can executecertain stimulation commands 322 using specified output systems 390 topreempt planned maneuvers, providing riders with anticipatorystimulation. Additionally or alternatively, the sensory stimulationsystem 300 can execute certain stimulation commands 322 using particularoutput systems 390 as maneuvers are being performed. For example, thesensory stimulation system 300 can execute visual stimulation commands322 for a swerve maneuver 1-2 seconds prior to the AV control system 305performing the maneuver (e.g., by generating a visual output indicatingthe swerve maneuver on a light bar). As the swerve maneuver is executed,the sensory stimulation system 300 can execute stimulation commands 322that roll the seats into the turns. Thus, any combination of preemptiveand reactive stimulation commands 322 can be executed to providestimulation outputs through any combination of the output systems 390for each respective maneuver performed by the AV 200. Such maneuvers cancomprise any high or low level maneuver involving acceleration, braking,and turning the AV 200. As described herein, the stimulation commands322 may be executed preemptively or reactively utilizing the visualsystem 393 (e.g., to provide light bar outputs or generate indicator onthe display) (556), the seat response systems 397 to provide hapticfeedback and/or pitch, roll, yaw sensations (557), the audio system 395to provide speech and/or tonal stimulations indicating each maneuver(558), and/or the airflow system 399 to provide air pressure stimulationbased on the maneuvers (559).

In certain aspects, the sensory stimulation system 300 can provide foruser interactivity to enable a rider to control certain parameters ofthe output systems 390. For example, a rider may access a menu on a userinterface of the display that can provide the adjustable parameters foreach of the output systems 390. Accordingly, the sensory stimulationsystem 300 can receive user inputs to adjust the parameters of ordeactivate some or all of the output systems 390 (560). In response tothe user inputs, the sensory stimulation system 300 can adjust theparameters of the specified output systems 390 or deactivate selectedsystems 390 accordingly (565). For example, a rider may wish to decreasethe sensitivity of the seat response system 397 to reduce the amount ofpitch or roll of the seats when stimulation commands 322 are executed.The rider may wish to increase the intensity of the haptic system toprovide stronger vibrational feedback indicating upcoming maneuvers. Asanother example, the rider may wish to deactivate the audio outputsaltogether to, for example, listen to music or a radio program. Thesensory stimulation system 300 can provide a user interface to enablethe rider to make adjustments and/or deactivate one or more of theoutput systems 390 accordingly.

Furthermore, the above-discussed operations in connection with FIGS. 5Aand 5B make be repeated and/or performed dynamically for each maneuverperformed by the AV control system 305 and/or each motion actionexperienced by the AV 200.

Hardware Diagram

FIG. 6 is a block diagram illustrating a computer system upon whichexamples described herein may be implemented. For example, the intentionsignaling system 235, 300 shown and described with respect to FIGS. 2and 3 may be implemented on the computer system 600 of FIG. 6. Thecomputer system 600 can be implemented using one or more processors 604,and one or more memory resources 606. In the context of FIGS. 2 and 3,the sensory stimulation system 235, 300 can be implemented using one ormore components of the computer system 600 shown in FIG. 6.

According to some examples, the computer system 600 may be implementedwithin an autonomous vehicle with software and hardware resources suchas described with examples of FIGS. 1 through 3. In an example shown,the computer system 600 can be distributed spatially into variousregions of the autonomous vehicle, with various aspects integrated withother components of the autonomous vehicle itself. For example, theprocessors 604 and/or memory resources 606 can be provided in the trunkof the autonomous vehicle. The various processing resources 604 of thecomputer system 600 can also execute stimulation instructions 612 usingmicroprocessors or integrated circuits. In some examples, thestimulation instructions 612 can be executed by the processing resources604 or using field-programmable gate arrays (FPGAs).

In an example of FIG. 6, the computer system 600 can include a localcommunication interface 650 (or series of local links) to vehicleinterfaces and other resources of the autonomous vehicle (e.g., thecomputer stack drives). In one implementation, the communicationinterface 650 provides a data bus or other local links toelectro-mechanical interfaces of the vehicle, such as wireless or wiredlinks to the AV control system 220, 305.

The memory resources 606 can include, for example, main memory, aread-only memory (ROM), storage device, and cache resources. The mainmemory of memory resources 606 can include random access memory (RAM) orother dynamic storage device, for storing information and instructionswhich are executable by the processors 604. The processors 604 canexecute instructions for processing information stored with the mainmemory of the memory resources 606. The main memory 606 can also storetemporary variables or other intermediate information which can be usedduring execution of instructions by one or more of the processors 604.The memory resources 606 can also include ROM or other static storagedevice for storing static information and instructions for one or moreof the processors 604. The memory resources 606 can also include otherforms of memory devices and components, such as a magnetic disk oroptical disk, for purpose of storing information and instructions foruse by one or more of the processors 604.

According to some examples, the memory 606 may store a plurality ofsoftware instructions including, for example, stimulation instructions612. The stimulation instructions 612 may be executed by one or more ofthe processors 604 in order to implement functionality such as describedwith respect to the sensory stimulation system 235, 300 of FIGS. 2 and3.

In certain examples, the computer system 600 can receive sensor data 662over the communication interface 650 from various AV subsystems 660(e.g., the AV control system 220, 305 or an on-board computer 210respectively). Additionally or alternatively, the computer system canreceive action data 664 corresponding to acceleration, braking, andsteering inputs to be performed by the AV control system 220, 305. Inexecuting the stimulation instructions 612, the processing resources 604can monitor the sensor data 662 and/or the action data 664 and generatestimulation outputs to the interior output systems 620 of the AV 200 inaccordance with examples described herein.

It is contemplated for examples described herein to extend to individualelements and concepts described herein, independently of other concepts,ideas or systems, as well as for examples to include combinations ofelements recited anywhere in this application. Although examples aredescribed in detail herein with reference to the accompanying drawings,it is to be understood that the concepts are not limited to thoseprecise examples. As such, many modifications and variations will beapparent to practitioners skilled in this art. Accordingly, it isintended that the scope of the concepts be defined by the followingclaims and their equivalents. Furthermore, it is contemplated that aparticular feature described either individually or as part of anexample can be combined with other individually described features, orparts of other examples, even if the other features and examples make nomentioned of the particular feature. Thus, the absence of describingcombinations should not preclude claiming rights to such combinations.

What is claimed is:
 1. A sensory stimulation system for an autonomousvehicle (AV), the sensory stimulation system comprising: at least onedisplay unit within an interior of the AV; one or more processors; andone or more memory resources storing instructions that, when executed bythe one or more processors, causes the sensory stimulation system to:determine a set of maneuvers to be executed by the AV; based on eachrespective maneuver of the set, determine a set of visual stimulationoutputs to provide a passenger of the AV with visual indications of therespective maneuver; and prior to the AV executing the respectivemaneuver, preemptively display the set of visual stimulation outputs onthe at least one display unit.
 2. The sensory stimulation system ofclaim 1, wherein the set of maneuvers comprise acceleration, braking,and change of direction actions executed by the AV.
 3. The sensorystimulation system of claim 1, wherein the executed instructions causethe sensory stimulation system to determine the set of maneuvers byreceiving action data from a control system of the AV, the action dataindicating an immediate action plan corresponding to the set ofmaneuvers to be executed by the AV.
 4. The sensory stimulation system ofclaim 1, wherein the executed instructions further cause the sensorystimulation system to: display a live presentation on the at least onedisplay unit, the live presentation presenting the AV operating along acurrent route.
 5. The sensory stimulation system of claim 4, wherein thelive presentation comprises LIDAR data from at least one LIDAR sensor ofthe AV, the LIDAR data providing a sensor view of a surroundingenvironment of the AV.
 6. The sensory stimulation system of claim 4,wherein the live presentation comprises live video data from one or morecameras of the AV.
 7. The sensory stimulation system of claim 5, whereinthe executed instructions cause the sensory stimulation system todisplay the live presentation by generating augmented reality featurespresented on the live video data, the augmented reality featuresrepresenting actual features in a surrounding environment of the AV. 8.The sensory stimulation system of claim 7, wherein the augmented realityfeatures comprise (i) an augmented reality representation of the AVtraveling along a current route, and (ii) generated representations ofproximate external entities to the AV.
 9. The sensory stimulation systemof claim 8, wherein the proximate external entities comprise proximatepedestrians, proximate bicyclists, and proximate vehicles to the AV. 10.The sensory stimulation system of claim 4, wherein the live presentationincludes a live map indicating route information of a current routetraveled by the AV.
 11. An autonomous vehicle (AV) comprising: a sensorsystem to generate sensor data indicating a situational environment ofthe AV; at least one display unit within an interior of the AV;acceleration, steering, and braking systems; and a control systemcomprising one or more processing resources executing instructions thatcause the control system to: determine a set of maneuvers to execute toautonomously control the acceleration, steering, and braking systems;based on each respective maneuver of the set, determine a set of visualstimulation outputs to provide a passenger of the AV with visualindications of the respective maneuver; and prior to the executing therespective maneuver, preemptively display the set of visual stimulationoutputs on the at least one display unit.
 12. The AV of claim 11,wherein the set of maneuvers comprise acceleration, braking, and changeof direction actions executed by the acceleration, steering, and brakingsystems.
 13. The AV of claim 11, wherein the executed instructions causethe control system to (i) generate action data indicating an immediateaction plan corresponding to the set of maneuvers, and (ii) determinethe set of visual stimulation outputs based on the action data.
 14. TheAV of claim 11, wherein the executed instructions further cause thecontrol system to: display a live presentation on the at least onedisplay unit, the live presentation presenting the AV operating along acurrent route.
 15. The AV of claim 14, wherein the live presentationcomprises LIDAR data from at least one LIDAR sensor of the sensorsystem, the LIDAR data providing a sensor view of a surroundingenvironment of the AV.
 16. The AV of claim 14, wherein the livepresentation comprises live video data from one or more cameras of thesensor system.
 17. The AV of claim 16, wherein the executed instructionscause the control system to display the live presentation by generatingaugmented reality features presented on the live video data, theaugmented reality features representing actual features in a surroundingenvironment of the AV.
 18. The AV of claim 17, wherein the augmentedreality features comprise (i) an augmented reality representation of theAV traveling along a current route, and (ii) generated representationsof proximate external entities to the AV.
 19. The AV of claim 18,wherein the proximate external entities comprise proximate pedestrians,proximate bicyclists, and proximate vehicles to the AV.
 20. Anon-transitory computer readable medium storing instructions forproviding sensory stimulation for one or more riders to indicationmotion of an autonomous vehicle (AV), wherein the instructions, whenexecuted by one or more processors, cause the one or more processors to:determine a set of maneuvers to be executed by the AV; based on eachrespective maneuver of the set, determine a set of visual stimulationoutputs to provide a passenger of the AV with visual indications of therespective maneuver; and prior to the AV executing the respectivemaneuver, preemptively display the set of visual stimulation outputs onthe at least one display unit.